Distributed file systems offer many compelling advantages in establishing high performance computing environments. One example is the ability to easily expand, even at large scale. In some distributed file systems, specifically those that use distributed locking of files and metadata structures to process operations, each node may coordinate a small subset of the overall number of locks on the cluster of nodes. If one node then wishes to assert a lock on a file or structure that is coordinated by another node, the node must send a communication packet to the coordinator node that asks for the lock to be established and then receive confirmation from the coordinator when the request been granted. It can be appreciated that in large scale distributed file systems, that the number of locks that are managed in this manner can reach tens to hundreds of million or more depending on the scale of the distributed system.
When a node or group of nodes is added to a cluster of nodes, or when a node or nodes is removed from a cluster of nodes, the resources of the cluster can change dramatically. In the event where a node leaves the cluster, any locks that were previously coordinated by that node must be redistributed. In one example, the locks can be redistributed by a coordinator node assigned among the surviving nodes. It can be appreciated, however, that in many distributed file systems coordinators are not assigned deterministically. When a node has become the new coordinator for a lock or a group of locks, at first, it will have minimal knowledge about which other nodes, e.g. initiator nodes, have locks asserted on the structures the coordinator node is now coordinating. Thus, the coordinator must either ask for or wait passively for and then receive, updates from all initiators nodes that effectively reassert their locks. With potentially millions of locks to assert, the amount of backplane communication among the cluster can be substantial. It can be further appreciated that the during the lock reconstruction process after a group change, the entire distributed file system is locked down until the group change process is complete, suspending any pending operations and likely penalizing performance to any clients connected to the distributed file system. Thus there exists a need to make the group change process more efficient to reduce any downtime by the distributed file system during a group change.